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Description 

BACKGROUND OF THE INVENTION 

[0001] The present invention relates to digital televi- 5 
sion, and more particularly to a method for video inser- 
tion or stream switching in progressively refreshed video 
streams. 

[0002] Commercial insertion, promo insertion, studio 
routing, camera switching, tape editing, and the like are 
basic operations in television production and broadcast- 
ing. In present day analog systems, switching takes 
place in the vertical interval of the video signal. With the 
advent of digital television as implemented, for example, 
in the Motion Picture Experts Group (MPEG) standards, 
switching or "splicing" with compressed bitstreams is 
very complicated. In order to facilitate such operations, 
the MPEG-2 standard defines "splice points" where 
point signals (e.g., a commercial) can be substituted for 
another signal (e.g., a main program) in a video stream. 
[0003] The Society of Motion Picture and Television 
Engineers (SMPTE) is currently working on a draft 
standard for splicing MPEG-2 transport streams. See, 
for example, Proposed SMPTE Standard PT20.02/010 
"Splice Points for MPEG-2 Transport Streams", Second 
Draft, July 1997. 

[0004] Further information about splicing of com- 
pressed video can be found in US patent publication 
5,602,592. 

[0005] There are various reasons why splicing MPEG 
video is more difficult than switching uncompressed vid- 
eo. In order to explain this, it is necessary to understand 
some basics about the MPEG transport stream. MPEG 
gets much of its compression capability by sending only 
the changes between different video frames. A first type 
of frame, known as a prediction or "P" frame, is used by 
the decoder to predict the frame from a previous "P" 
frame or from an intra-coded T frame in the video 
stream. The stream merely carries "fine tuning" informa- 
tion to correct errors from an approximate prediction. An 
I frame is compressed without motion prediction. Thus, 
a full frame can be reconstructed from an I frame without 
reference to any other frame. In this manner, errors in 
the prior frame predictions will be eliminated once an I 
frame arrives and is decoded. 

[0006] Bidirectional "B" frames are like P frames, ex- 
cept that the prediction is made not only from the previ- 
ous I or P frame, but also from the next one. For pur- 
poses of splicing, this means that while it is possible to 
leave an old stream after any P or I frame, it is not pos- 
sible to leave the stream after a B frame since that would 
cut off the backward prediction reference frame for that 
B frame. From the point of view of the bitstream, this 
means that a first video stream may only be left after a 
P or I frame and all of the B frames (if any) that imme- 
diately follow it have passed. 

[0007] Matters are further complicated when the com- 
pressed MPEG bitstream is progressively refreshed. In 



progressively refreshed streams, such as taught in com- 
monly assigned US patent 5,057,916 to Krause et al. 
entitled "Method and Apparatus for Refreshing Motion 
Compensated Sequential Video Images" or US patent 
5,568,200 (Pearlstein et al., 'Method and Apparatus for 
improved Video Display of progressively refreshed cod- 
ed Video"), sequential video images are refreshed one 
region at a time. A different region in each of the plurality 
of video images is communicated without compression 
during a refresh cycle. In this manner, an image area 
defined by the region is progressively refreshed by the 
non-compressed regions during the refresh cycle. Com- 
pression of video images may be controlled to prevent 
data contained in regions not yet refreshed during a cur- 
rent refresh cycle from corrupting data contained in re- 
gions that have been refreshed during the current re- 
fresh cycle. 

[0008] For commercial insertion or switching purpos- 
es, compressed video from a first source is replaced 
with compressed video from a second source at a pre- 
defined insertion point in the video data stream. If the 
video from either or both of the first and second source 
is progressively refreshed, then progressive refresh re- 
lated artifacts will appear at one or both of the bounda- 
ries between the first and second source video. 
[0009] It would be advantageous to provide a method 
for providing splicing of progressively refreshed video 
streams that minimizes noticeable artifacts. The present 
invention provides such a method. 

SUMMARY OF THE INVENTION 

[0010] In accordance with the appended claim 1, a 
method is provided for replacing compressed video from 
a first source with compressed video from a second 
source , e.g., at a predefined insertion point in a video 
data stream. A determination is made as to whether the 
video from the second source is progressively refreshed 
video. The determination can be a real time determina- 
tion (e.g., based on testing the second source) or can 
result from a priori knowledge of the second source. If 
the video from the second source is progressively re- 
freshed, then it is decompressed sufficiently prior in time 
to the replacement of the first source video to enable the 
recovery and recompression of a first replacement video 
frame from said second source before replacement 
commences. It is noted that for purposes of present dis- 
closure, the "first replacement video frame" is the first 
frame in encoding order of the replacement video 
stream, which may differ from the display order, as well 
known in the art. 

[0011] The recovered first video frame is recom- 
pressed as an intra-coded (I) frame. Then, insertion of 
the video from the second source is commenced (e.g., 
atthe predefined insertion point) using the I frame. Since 
the I frame comprises all of the information necessary 
to reconstruct a full frame of data and is not progressive- 
ly refreshed, it enables a clean switch to be made from 
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the video data from the first source to the video data 
from the second source. 

[0012] The insertion point can, for example, indicate 
the commencement of a commercial break in the video 
stream. In this instance, the second source provides a 
commercial for insertion into the commercial break. At 
the conclusion of the commercial break, video from the 
first source recommences at a second insertion point in 
the data stream. As with the first splice, it is necessary 
to determine if the video from the first source is progres- 
sively refreshed video. If so, that video is decompressed 
sufficiently prior to the second insertion point to enable 
the recovery and recompression of a first video frame 
(in encoding order) from the first source to be inserted 
at the second insertion point. The recovered first video 
frame from the first source is then recompressed to pro- 
duce an I frame. Insertion of the video from the first 
source at the second insertion point commences using 
the I frame produced from the first source. 
[0013] A method is also provided for replacing com- 
pressed video from a first source with substitute pro- 
gressively refreshed compressed video (e.g., at a pre- 
defined insertion point) in a video data stream. The sub- 
stitute video (e.g., a commercial) is stored with a first 
frame thereof encoded as an intra-coded (I) frame. The 
insertion of the substitute video is commenced (e.g., at 
the predefined insertion point) using the I frame as the 
first inserted frame. When the compressed video from 
the first source is progressively refreshed and is to be 
recommenced at a second insertion point in the data 
stream following the first insertion point (e.g., at the end 
of a commercial break) the video from the first source 
must be processed to prevent progressive refresh relat- 
ed artifacts. In particular, the video from the first source 
is decompressed sufficiently prior to the second inser- 
tion point to enable the recovery and recompression of 
a first video frame from the first source to be inserted at 
the second insertion point. The recovered first video 
frame from the first source is recompressed to produce 
an l-frame. Insertion of the video from the first source is 
commenced at the second insertion point using the I 
frame produced from the first source. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0014] 

FIG. 1 is a diagram illustrating first and second data 
streams S1, S2, respectively, together with a third 
data stream S3 comprising stream S1 into which a 
portion of stream S2 has been inserted; and 
FIG. 2 is a diagram illustrating a data stream S4 cre- 
ated by switching from data stream S1 to data 
stream S2. 

DETAILED DESCRIPTION OF THE INVENTION 
[0015] The present invention provides a method for 



inserting commercials or other video segments into pro- 
gressively refreshed bitstreams and for switching from 
one bitstream to another when any of the bitstreams is 
progressively refreshed. 

5 [001 6] FIG. 1 illustrates a first compressed digital vid- 
eo stream 10 (stream S1). The video stream 10 has a 
first insertion point t1 at which a commercial break com- 
mences. The commercial break terminates at insertion 
point t2. It is noted that the insertion points may or may 

10 not comprise predetermined points in the video stream. 
Since it is more typical to insert a video segment such 
as a commercial at a predefined time, it is expected that 
predetermined insertion points will more often be used 
with the present invention. However, the use of prede- 

15 termined insertion points is not required in accordance 
with the invention. For example, a local television studio 
may desire to insert a local segment into a main video 
stream at an arbitrary time, in which case the invention 
may be implemented without using a predetermined in- 

20 sertion point for the replacement of the main video with 
the replacement video. 

[0017] As indicated in FIG. 1, a commercial or other 
video segment S1c is inserted in the commercial break 
defined by insertion points t1 and t2. In particular, the 
25 inserted video S1c commences at insertion point t1 and 
terminates at insertion point t2, where the main video 
program carried by S1 restarts at the termination of the 
commercial break. 

[001 8] A similar video stream 1 2 (stream S2) compris- 

30 es a main program S2 into which a commercial S2c is 
inserted. It is noted that the commercials S1c and S2c 
may be part of the streams S1 and S2, respectively, in 
which case they need not be separately inserted into the 
main stream. However, in the event that the commer- 

35 cials (or other video segments that are inserted) are 
stand alone video segments, they must be inserted into 
the main stream at the appropriate insertion point t1 and 
terminated at the insertion point t2 when the main 
stream recommences. 

40 [0019] Video stream 14 (stream S3) represents a 
classic case of video splicing, in which part of stream 
S1 is replaced by S2c. As indicated above, S2c can be 
part of stream S2 or may be a stand alone commercial 
or other stand alone video segment. If S1 and/or S2 are 

45 progressively refreshed video data streams, then pro- 
gressive refresh related artifacts will occur at one or both 
the boundaries of S1 and S2c. Such refresh related ar- 
tifacts can last as long as one second or more, depend- 
ing upon the refresh rate. Thus, when a commercial is 

so switched in and out one would see the artifacts every 
time the switch is made. To prevent such artifacts while 
switching the commercial into a main data stream (i.e., 
splicing out of the main stream and cutting the commer- 
cial in), the commercial can be compressed in I, P, B 

55 frame mode. However, when the commercial is over and 
the progressively refreshed main stream is switched 
back in, the progressive refresh artifacts wilt stilt occur. 
It is noted that the video frames in the stream illustrated 
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in FIG. 1 are in encoding order, which is different from 
the frame display order. When the frames are encoding 
order, the first frame of the switched-in segment will be 
an I frame. 

[0020] For purposes of the present disclosure, the 
terms "frame" and "video frame" are used in a generic 
sense to mean a portion of a video signal or stream from 
which other portions are reconstructed using, for exam- 
ple, an l-frame, P-frame or B-frame. Thus, although typ- 
ically a "frame" will be a "full" video frame, it is not in- 
tended to limit the meaning thereof to a full video frame 
for purposes of the present disclosure, since implemen- 
tations may exist or be developed wherein, e.g., sub- 
frame, field, subfield, or other processing is used. With 
respect to processing of video on a field level instead of 
a frame level, it should be appreciated that l-fields, P- 
fields and B-fields may be used. In such cases, these 
fields are equivalent to and are intended to be covered 
by the broad terms "frame" and "video frame" as used 
in the frame processing of video signals. 
[0021 ] The present invention overcomes the problem 
of progressive refresh related artifacts by commencing 
processing of a frame to be inserted prior to the insertion 
time. First, the video to be inserted is decompressed suf- 
ficiently prior to the insertion point to enable the recovery 
and recompression of a first frame to be inserted. As 
noted above, the "first frame to be inserted" is the first 
frame in the encoding order and not necessarily the first 
frame in the display order. Once this frame has been 
decompressed, it is recompressed as an intra-coded (I) 
frame. Such intra-coded frames are able to be recon- 
structed without referring to any other frame, and there- 
fore are not subject to progressive refresh. After the re- 
covered first frame is recompressed as an I frame, it is 
used as the first frame inserted at the insertion point. 
[0022] As a first example, S1 may be a progressively 
refreshed compressed video stream whereas S2 is not 
progressively refreshed. If a sequence is not progres- 
sively refreshed then it is l-frame refreshed. In this, 
case, progressive refresh artifacts will occur in switching 
from S2 to S1 , but not while going from S1 to S2. There- 
fore, a switch can be made from St to S2 in order to 
insert S2c in a conventional manner as is done for I- 
frame refreshed cases. However, the technique of the 
present invention must be used to avoid progressive re- 
fresh artifacts when switching back from S2c to S1 . 
[0023] In accordance with the invention, when S2c is 
inserted, decoding of S1 commences at some time dt 
earlier than the insertion point t2. In other words, decod- 
ing of S1 will commence at time (t2-crt). Upon recovery 
of a first frame (in encoding order) of S1 to be inserted 
at the termination of S2c, this frame or "picture" is rec- 
ompressed as all intra-coded blocks, and the recom- 
pressed picture is inserted commencing at insertion 
point t2. The remainder of stream S1 can be the same 
as the original stream. 

[0024] In another example, both S1 and S2 are pro- 
gressively refreshed. In this case, when switching from 



6 

stream S1 to video segment S2c, decoding of S2c com- 
mences at some time oT earlier than insertion t1 , i.e., at 
(t1-d/). A recovered first frame (in encoding order) of 
S2c is recompressed as all I blocks. This first frame is 

5 inserted as the first frame of the video segment S2c at 
insertion point t1 . No special processing is required for 
the remainder of video segment S2c. When switching 
back from S2c to S1, the same technique used in the 
previous example is followed; i.e., the first frame (in en- 

10 coding order) of stream S1 to be inserted at insertion 
point t2 is recovered and recompressed as all I blocks 
for insertion at t2. 

[0025] FIG. 2 illustrates a special case referred to as 
"switching." In this case, a switch from one stream (e. 
is g. ( a feed from a first camera) to another stream (e.g., 
a feed from a second camera) is made. As illustrated in 
FIG. 2, stream 16 (stream S4) is created by switching 
S1 to stream S2 at insertion point (or "switching point") 
t1. In the event that S1 is progressively refreshed but 
20 S2 is not, no special processing is required. However, 
in the case where S2 is progressively refreshed (regard- 
less of whether S1 is progressively refreshed) decoding 
of S1 will be commenced at a time dt prior to the switch- 
ing point t1 so that a first frame thereof (in encoding or- 
25 der) can be recompressed as an I frame for use as the 
first frame when the streams are switched at switching 
point t1. 

[0026] The times df and df'are established by know- 
ing how many frames it takes to complete the progres- 
30 sive refresh of a picture. For example, if the progressive 
refresh is completed within fifteen frames, c/f and c/f will 
be a period of time at least equivalent to fifteen frame 
times, and more typically will be greater than this (e.g., 
thirty frames) to be certain that enough time is provided 
35 to decode and recompress the necessary first frame of 
the inserted video portion. 

[0027] It should now be appreciated that the present 
invention avoids progressive refresh artifacts by starting 
to decode a main stream, which is progressively re- 
40 freshed, several frames before an inserted video portion 
(e.g., a commercial) is over. At the time when the com- 
mercial is over, the decoded picture (e.g., P picture) of 
the main stream is captured and recompressed as an I 
frame. The main stream is then switched-in commenc- 
es jng with that I frame. The rest of the frames in the main 
stream can be progressively refreshed frames. It is not- 
ed that there may still be some artifacts due to the fact 
that the I frame is not the true reference frame used to 
encode the main stream. However, such artifacts are 
so generally significantly less objectionable than the pro- 
gressive refresh artifacts that would otherwise occur. 
[0028] In addition, it is also possible to store the com- 
mercial or other inserted video segment such that only 
the first frame of the segment is an I frame and the rest 
55 of the frames are progressively refreshed. In this man- 
ner, a progressively refreshed commercial can be used 
without detrimental progressive refreshed artifacts at 
the time the commercial is switched in to the video 
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stream. 

[0029] Although the invention has been described in 
connection with various specific embodiments, it should 
be appreciated that numerous adaptations and modifi- 
cations may be made thereto without departing from the 
scope of the invention as set forth in the claims. 



Claims 

1. A method for replacing compressed video from a 
first source with compressed video from a second 
source in a video data stream, comprising the steps 
of: 

(a) decompressing the video from said second 
source sufficiently prior to the replacement of 
the first source video to enable the recovery of 
a first replacement video frame from said sec- 
ond source before replacement commences; 

(b) re-compressing the recovered first replace- 
ment video frame as an intra-coded (I) frame; 
and 

(c) commencing the replacement of the first 
source video with the second source video us- 
ing said I frame; 

characterized by 

determining if the video from said sec- 
ond source is progressively refreshed video, and if 
so: 
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(a) decompressing the video from said first 
source sufficiently prior to said second in- 
sertion point to enable the recovery of a 
first video frame from said first source to be 
inserted at said second insertion point; 

(b) re-compressing the recovered first vid- 
eo frame from said first source to produce 
an intra-coded (I) frame; and 

(c) commencing the insertion of the video 
from said first source at said second inser- 
tion point using the I frame produced from 
said first source. 



15 Patentanspruche 

1 . Ein Verfahren zum Ersetzen komprimierten Videos 
von einer ersten Quelle mit komprimiertem Video 
von einer zweiten Quelle in einem Videodaten- 
20 strom, das folgende Schritte beinhaltet: 

(a) Dekomprimieren des Videos von der zwei- 
ten Quelle in ausreichendem Zeitraum vor dem 
Ersetzen des Videos der ersten Quelle, urn die 

25 Wiederherstellung eines ersten Ersatzvideob- 

ilds von der zweiten Quelle einzuschalten, be- 
vor das Ersetzen beginnt; 

(b) Wiederkomprimieren des wiederhergestell- 
30 ten ersten Ersatzvideobilds als intracodiertes 

Bild (l-Bild); und 



recovering said first replacement video frame 
from the number of frames it takes to complete 
the progressive refresh of a frame for re-com- 
pression as said intra-coded (I) frame. 

A method in accordance with claim 1 wherein the 
replacement of the first source video with the sec- 
ond source video commences with the insertion of 
said I frame at a predefined insertion point in said 
video data stream. 

A method in accordance with claim 2 wherein said 
insertion point indicates the commencement of a 
commercial break in said video stream, and said 
second source provides a commercial for insertion 
into said commercial break. 

A method in accordance with claim 3 wherein at the 
conclusion of said commercial break, video from 
said first source recommences at a second inser- 
tion point in said data stream, comprising the further 
steps of: 

determining if the video from said first source 
is progressively refreshed video, and if so: 
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(c) Beginnen mit dem Ersetzen des Videos der 
ersten Quelle durch das Video der zweiten 
Quelle unter Verwendung des l-Bilds; 

gekennzeichnet durch das 

Bestimmen, ob das Video von der zweiten 
Quelle progressiv aufgefrischtes Video ist, und 
wenn ja: 

Wiederherstellen des ersten Ersatzvideobilds 
aus der Anzahl von Bildern, die benotigt wer- 
den, urn die progressive Auffrischung eines 
Bilds zur Wiederkomprimierung als intracodier- 
tes Bild (l-Bild) zu vollenden. 

Verfahren gemaG Anspruch 1 , wobei das Ersetzen 
des Videos der ersten Quelle durch das Video der 
zweiten Quelle mit der Einfugung des l-Bilds an ei- 
nem vorgegebenen Einfugepunkt in dem Videoda- 
tenstrom beginnt. 

Verfahren gemafc Anspruch 2, wobei der Einfuge- 
punkt den Beginn eines Werbebtocks in dem Vide- 
ostrom angibt und die zweite Quelle eine Werbeein- 
btendung zur Einfugung in den Werbeblock bereit- 
stellt. 
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4. Verfahren gemaB Anspruch 3, wobei beim Ab- 
schluss des Werbeblocks Video von der ersten 
Quelle an einem zweiten Einfiigepunkt in dem Da- 
tenstrom wieder beginnt, das die folgenden weite- 
ren Schritte beinhaltet: s 



1. Un procede destine a remplacer de la video com- 
pressee d'une premiere source par de la video com- 
pressee d'une deuxieme source dans un flux de 
donnees video, comportant les etapes de : 35 



caracterise par 

determiner si la video de ladite deuxieme source est 
de la video progressivement rafrafchie, et si c'est le 55 
cas : 

recuperer ladite premiere trame video de rem- 



placement a partir du nombre de trames que 
cela prend pour terminer le rafraTchissement 
progressif d'une trame pour recompression en 
tant que dite trame (I) intra. 

2. Un procede conformement a la revendication 1 
dans lequel le remplacement de la video de premie- 
re source par la video de deuxieme source com- 
mence avec 1'insertion de ladite trame I a un point 
d'insertion pr6d6fini dans ledit flux de donnees vi- 
deo. 

3. Un procede conformement a la revendication 2 
dans lequel ledit point d'insertion indique le com- 
mencement d'une pause publicitaire dans ledit flux 
video, et ladite deuxieme source fournit une publi- 
city pour insertion dans ladite pause publicitaire. 

4. Un procede conformement a la revendication 3 
dans lequel au terme de ladite pause publicitaire, la 
video de ladite premiere source recommence a un 
deuxieme point d'insertion dans ledit flux de don- 
nees, comportant les etapes supplementaires de : 

determiner si la video de ladite premiere source 
est une video progressivement rafraichie, et si 
c'est le cas : 

(a) decompresser la video de ladite pre- 
miere source de facon suffisamment prea- 
mble audit deuxieme point d'insertion pour 
permettre la recuperation d'une premiere 
trame video a partir de ladite premiere 
source devant etre inseree audit deuxieme 
point d'insertion ; 

(b) recompresser la premiere trame video 
r6cup6r£e de ladite premiere source pour 
produire une trame (I) intra ; et 

(c) commencer I'insertion de la video de la- 
dite premiere source audit deuxieme point 
d'insertion en utilisant la trame 1 produite 
a partir de ladite premiere source. 



Bestimmen, ob das Video von der ersten Quel- 
le progressiv aufgefrischtes Video ist, und 
wenn ja: 

w 

(a) Dekomprimieren des Videos von der er- 
sten Quelle in ausreichendem Zeitraum 
vor dem zweiten Einfiigepunkt, um die 
Wiederherstellung eines ersten Videobilds 
von der ersten Quelle einzuschalten, das '5 
an dem zweiten Einfiigepunkt eingefugt 
werden soil; 

(b) Wiederkomprimieren des wiederherge- 
stellten ersten Videobilds von der ersten 20 
Quelle, um ein intracodiertes Bild (l-Bild) 

zu erzeugen; und 

(c) Beginnen mit dem Einfiigen des Videos 
von der ersten Quelle an dem zweiten Ein- 25 
fiigepunkt unter Verwendung des l-Bilds, 
das von der ersten Quelle erzeugt wurde. 

Revendications 30 



(a) decompresser la video de ladite deuxieme 
source de facon suffisamment prealable au 
remplacement de la video de premiere source 
pour permettre la recuperation d'une premiere 40 
trame video de remplacement a partir de ladite 
deuxieme source avant que le remplacement 
commence ; 

(b) recompresser la premiere trame video de 45 
remplacement r6cuperee en tant que trame (I) 
intra ; et 

(c) commencer le remplacement de la video de 
premiere source par la video de deuxieme so 
source en utilisant ladite trame I ; 
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